{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Navigation\n",
    "\n",
    "---\n",
    "\n",
    "Congratulations for completing the first project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893)!  In this notebook, you will learn how to control an agent in a more challenging environment, where it can learn directly from raw pixels!  **Note that this exercise is optional!**\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "We begin by importing some necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np\n",
    "import time\n",
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/VisualBanana.app\"`\n",
    "- **Windows** (x86): `\"path/to/VisualBanana_Windows_x86/Banana.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/VisualBanana_Windows_x86_64/Banana.exe\"`\n",
    "- **Linux** (x86): `\"path/to/VisualBanana_Linux/Banana.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/VisualBanana_Linux/Banana.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/VisualBanana_Linux_NoVis/Banana.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/VisualBanana_Linux_NoVis/Banana.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `VisualBanana.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"VisualBanana.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "###env = UnityEnvironment(file_name=\"./VisualBanana_Linux/Banana.x86_64\", no_graphics=False)\n",
    "###env = UnityEnvironment(file_name=\"./VisualBanana_Linux/Banana.x86_64\", no_graphics=True)\n",
    "env = UnityEnvironment(file_name=\"./VisualBanana_Linux/Banana.x86_64\") #suggested by Udacity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "The simulation contains a single agent that navigates a large environment.  At each time step, it has four actions at its disposal:\n",
    "- `0` - walk forward \n",
    "- `1` - walk backward\n",
    "- `2` - turn left\n",
    "- `3` - turn right\n",
    "\n",
    "The environment state is an array of raw pixels with shape `(1, 84, 84, 3)`.  *Note that this code differs from the notebook for the project, where we are grabbing **`visual_observations`** (the raw pixels) instead of **`vector_observations`**.* A reward of `+1` is provided for collecting a yellow banana, and a reward of `-1` is provided for collecting a blue banana. \n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents in the environment\n",
    "print('Number of agents:', len(env_info.agents))\n",
    "\n",
    "# number of actions\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Number of actions:', action_size)\n",
    "\n",
    "# examine the state space\n",
    "state = env_info.visual_observations[0]\n",
    "\n",
    "print('States look like:')\n",
    "plt.imshow(np.squeeze(state, axis=0))\n",
    "plt.show()\n",
    "state_size = state.shape\n",
    "print('States have shape:', state.shape)\n",
    "#print(np.expand_dims(state, axis=4).shape) #this is unsqueeze\n",
    "state = state.reshape((-1,3,84,84))\n",
    "print('modified state is: ', state.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Take Random Actions in the Environment\n",
    "\n",
    "In the next code cell, you will learn how to use the Python API to control the agent and receive feedback from the environment.\n",
    "\n",
    "Once this cell is executed, you will watch the agent's performance, if it selects an action (uniformly) at random with each time step.  A window should pop up that allows you to observe the agent, as it moves through the environment.  \n",
    "\n",
    "Of course, you'll have to change the code so that the agent is able to use its experience to gradually choose better actions when interacting with the environment!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env_info = env.reset(train_mode=False)[brain_name] # reset the environment\n",
    "state = env_info.visual_observations[0]            # get the current state\n",
    "score = 0                                          # initialize the score\n",
    "while True:\n",
    "    action = np.random.randint(action_size)        # select an action\n",
    "    env_info = env.step(action)[brain_name]        # send the action to the environment\n",
    "    next_state = env_info.visual_observations[0]   # get the next state\n",
    "    reward = env_info.rewards[0]                   # get the reward\n",
    "    done = env_info.local_done[0]                  # see if episode has finished\n",
    "    score += reward                                # update the score\n",
    "    state = next_state                             # roll over the state to next time step\n",
    "    #print(done)\n",
    "    if done:                                       # exit loop if episode finished\n",
    "        break\n",
    "    \n",
    "print(\"Score: {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. It's Your Turn!\n",
    "\n",
    "Now it's your turn to train your own agent to solve the environment!  When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "#Training the agent\n",
    "from unityagents import UnityEnvironment\n",
    "import numpy as np\n",
    "import time\n",
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "import pickle\n",
    "from pixel_dqn_agent import Agent\n",
    "\n",
    "local_network_fn = './saved_agent/dqn_checkpoint_local.pth'\n",
    "target_network_fn = './saved_agent/dqn_checkpoint_target.pth'\n",
    "memory_buffer_fn = './saved_agent/memory_buffer'\n",
    "scores_fn = './saved_agent/scores.txt'\n",
    "#agent_fn = 'ddqn_checkpoint_agent.pth'\n",
    "def load_agent(agent):    \n",
    "    agent.qnetwork_local.load_state_dict(torch.load(local_network_fn, map_location=lambda storage, loc: storage))\n",
    "    agent.qnetwork_target.load_state_dict(torch.load(target_network_fn, map_location=lambda storage, loc: storage))\n",
    "    with open(scores_fn, \"rb\") as sf:\n",
    "        agent.scores = pickle.load(sf)\n",
    "    with open(memory_buffer_fn, \"rb\") as mf:\n",
    "        agent.memory.memory = pickle.load(mf)\n",
    "    #agent = Agent.load(agent_fn)\n",
    "    return agent\n",
    "\n",
    "def save_agent(agent):    \n",
    "    torch.save(agent.qnetwork_local.state_dict(), local_network_fn)\n",
    "    torch.save(agent.qnetwork_target.state_dict(), target_network_fn)\n",
    "    with open(scores_fn, \"wb\") as sf:\n",
    "        pickle.dump(agent.scores, sf, pickle.HIGHEST_PROTOCOL)\n",
    "    with open(memory_buffer_fn, \"wb\") as mf:\n",
    "        mem_to_save = deque(list(agent.memory.memory)[-50000:], maxlen=100000) #agent.memory.memory\n",
    "        pickle.dump(mem_to_save, mf, pickle.HIGHEST_PROTOCOL)\n",
    "    #agent.save(agent_fn)\n",
    "    return None\n",
    "\n",
    "def train_agent():\n",
    "    env = UnityEnvironment(file_name=\"./VisualBanana_Linux/Banana.x86_64\", seed=111)\n",
    "    brain_name = env.brain_names[0]\n",
    "    brain = env.brains[brain_name]\n",
    "\n",
    "    final_eps = 0.01\n",
    "    eps_start= 1.0 #0.01 #1.0\n",
    "    agent = Agent(num_input_chnl=11, action_size=4, seed=0) #create a new agent\n",
    "    #agent = load_agent(agent)\n",
    "  \n",
    "    def dqn(n_episodes=3000, max_t=1000, eps_start=eps_start, eps_end=final_eps, eps_decay=0.995):\n",
    "        \"\"\"Deep Q-Learning.\n",
    "\n",
    "        Params\n",
    "        ======\n",
    "            n_episodes (int): maximum number of training episodes\n",
    "            max_t (int): maximum number of timesteps per episode\n",
    "            eps_start (float): starting value of epsilon, for epsilon-greedy action selection\n",
    "            eps_end (float): minimum value of epsilon\n",
    "            eps_decay (float): multiplicative factor (per episode) for decreasing epsilon\n",
    "        \"\"\"\n",
    "        scores_window = deque(maxlen=100)  # last 100 scores\n",
    "        eps = eps_start                    # initialize epsilon\n",
    "        for i_episode in range(1, n_episodes+1):\n",
    "            env_info = env.reset(train_mode=True)[brain_name] # reset the environment\n",
    "            state = env_info.visual_observations[0]            # get the current state\n",
    "            #print(type(state))\n",
    "            state = state.reshape((-1,3,84,84))\n",
    "            #state = np.expand_dims(state, axis=0)\n",
    "            #print(state.shape)\n",
    "            #state = torch.from_numpy(state)\n",
    "            score = 0                                          # initialize the score\n",
    "            for t in range(max_t): #this could also be while True instead\n",
    "                aug_state = agent.augment_state(state)         # augment the state\n",
    "                action = agent.act(aug_state, eps)             # select an action using e-greedy policy\n",
    "                env_info = env.step(action)[brain_name]        # send the action to the environment\n",
    "                next_state = env_info.visual_observations[0]   # get the next state\n",
    "                next_state = next_state.reshape((-1,3,84,84))\n",
    "                reward = env_info.rewards[0]                   # get the reward\n",
    "                done = env_info.local_done[0]                  # see if episode has finished\n",
    "                agent.step(state, action, reward, next_state, done, is_training=True) #add to experience buffer and do the learning\n",
    "\n",
    "                score += reward                                # update the score\n",
    "                state = next_state                             # roll over the state to next time step\n",
    "                if done:                                       # exit loop if episode finished\n",
    "                    break                \n",
    "\n",
    "            scores_window.append(score)       # save most recent score\n",
    "            agent.scores.append(score)              # save most recent score\n",
    "            eps = max(eps_end, eps_decay*eps) # decrease epsilon\n",
    "            print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)), end=\"\")\n",
    "            if i_episode % 100 == 0:\n",
    "                print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)))\n",
    "                print(len(agent.memory.memory), agent.memory.memory.maxlen)\n",
    "            if i_episode % 500 == 0: #save weights every 500 episodes\n",
    "                save_agent(agent)\n",
    "            if np.mean(scores_window)>=17.0:\n",
    "                print('\\nEnvironment solved in {:d} episodes!\\tAverage Score: {:.2f}'.format(i_episode-100, np.mean(scores_window)))\n",
    "                save_agent(agent)\n",
    "                break\n",
    "        save_agent(agent) #save at the end\n",
    "\n",
    "        return agent.scores\n",
    "\n",
    "\n",
    "    strt = time.time()\n",
    "    scores = dqn()\n",
    "    print('\\nTraining Time is {}'.format(time.time()-strt))\n",
    "    env.close()\n",
    "\n",
    "#     # plot the scores\n",
    "#     fig = plt.figure()\n",
    "#     ax = fig.add_subplot(111)\n",
    "#     plt.plot(np.arange(len(scores)), scores)\n",
    "#     plt.ylabel('Score')\n",
    "#     plt.xlabel('Episode #')\n",
    "#     plt.show()\n",
    "\n",
    "\n",
    "#Train the agent\n",
    "print('Starting Agent Training:')\n",
    "train_agent()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting Agent Testing:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: BananaBrain\n",
      "        Number of Visual Observations (per agent): 1\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 0\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: discrete\n",
      "        Vector Action space size (per agent): 4\n",
      "        Vector Action descriptions: , , , \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 1: 11.0\n",
      "Mean Score out of 1 episodes is 11.0\n"
     ]
    }
   ],
   "source": [
    "#Testing the agent\n",
    "from unityagents import UnityEnvironment\n",
    "import numpy as np\n",
    "import time\n",
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "import pickle\n",
    "from pixel_dqn_agent import Agent\n",
    "\n",
    "local_network_fn = './saved_agent/dqn_checkpoint_local.pth'\n",
    "def test_agent(num_episodes=10):\n",
    "    #env = UnityEnvironment(file_name=\"./VisualBanana_Linux/Banana.x86_64\", seed=111)\n",
    "    env = UnityEnvironment(file_name=\"./VisualBanana_Linux/Banana.x86_64\")\n",
    "    brain_name = env.brain_names[0]\n",
    "    brain = env.brains[brain_name]\n",
    "\n",
    "    scores = []\n",
    "    final_eps=0.01    \n",
    "    agent = Agent(num_input_chnl=11, action_size=4, seed=0) #create a new agent\n",
    "    agent.qnetwork_local.load_state_dict(torch.load(local_network_fn, map_location=lambda storage, loc: storage)) #load the weights\n",
    "    \n",
    "    for i_episode in range(1,num_episodes+1):\n",
    "        env_info = env.reset(train_mode=False)[brain_name] # reset the environment\n",
    "        state = env_info.visual_observations[0]            # get the current state\n",
    "        state = state.reshape((-1,3,84,84))\n",
    "        score = 0                                          # initialize the score\n",
    "        while True:\n",
    "            aug_state = agent.augment_state(state)         # augment the state\n",
    "            action = agent.act(aug_state, final_eps)             # select an action using e-greedy policy\n",
    "            env_info = env.step(action)[brain_name]        # send the action to the environment\n",
    "            next_state = env_info.visual_observations[0]   # get the next state\n",
    "            next_state = next_state.reshape((-1,3,84,84))\n",
    "            reward = env_info.rewards[0]                   # get the reward\n",
    "            done = env_info.local_done[0]                  # see if episode has finished\n",
    "            agent.step(state, action, reward, next_state, done, is_training=False) #only add to experience buffer and don't do learning\n",
    "\n",
    "            score += reward                                # update the score\n",
    "            state = next_state                             # roll over the state to next time step\n",
    "            if done:                                       # exit loop if episode finished\n",
    "                scores.append(score)\n",
    "                print('Episode {}: {}'.format(i_episode, score))\n",
    "                break\n",
    "    env.close()\n",
    "    return np.mean(scores)\n",
    "\n",
    "print('Starting Agent Testing:')\n",
    "num_episodes=1 #100\n",
    "mean_score = test_agent(num_episodes)\n",
    "print(\"Mean Score out of {} episodes is {}\".format(num_episodes, mean_score))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEKCAYAAAASByJ7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztnXd8FVXax39PEgKhhkCAUEIQkCIoYARpKkUB0RV39xV1XbvYy77b4q6vdXVZdxXLqohr1wV7wSAKKNIUCb330CGhh5J+3j9m5mbuvTP3zsyddu99vp9PPpl75syc58ycOc8pz3kOCSHAMAzDMCleC8AwDMP4A1YIDMMwDABWCAzDMIwMKwSGYRgGACsEhmEYRoYVAsMwDAOAFQLDMAwjwwqBYRiGAcAKgWEYhpFJ81oAM7Rs2VLk5eV5LQbDMExcsXTp0oNCiOxo8eJKIeTl5aGoqMhrMRiGYeIKItphJB4PGTEMwzAAWCEwDMMwMqwQGIZhGACsEBiGYRgZVggMwzAMAFYIDMMwjAwrBIZhGAYAKwQmASivqsHHS3eDt4NlmNiIq4VpDKPFs7M2Ycq8bWiWUQ8X92zttTgME7dwD4GJe0rLKgAAZeVVHkvCMPENKwQmYeARI4aJDVYITNxDXgvAMAkCKwSGYRgGACsEhmEYRoYVApMw8BQCw8QGKwTGNNU1tbjg6e8xc81+r0WR8MEkwl8/W42CT1Z5LYZv5HCai5/9AZ8u2+21GDFTVVOLoU9/h2/W+uNbYoXAmObIqSrsPHwKD32+2mtRgvByYdr7i3di2pJdnqXvNzmcZnPJCfzvhyu9FiNmjpysxK7Dp/HQ52u8FgUAKwSGYeKMRFyR7pcssUJgLOOXQswwjD2wQmDiHvLDJALjGtwQcQ7HFQIRdSCi74loHRGtJaL75fAsIppFRJvl/82dloVJbLieYOKOQFvGH6XXjR5CNYDfCyF6AjgfwN1E1BNAAYA5QoiuAObIv5k4gLhBzniIP6rOxMRxhSCE2CeEWCYflwFYD6AdgCsAvC1HexvAOKdlYZxhz9HTOK7jWE7r3LHTVdh79HRQ2Ib9x22RpaZWYPOBsrDw8qoafLt2f2BCcvOBMtTUulO11NYKbNKQKVa0nqPXCCGweNsh7Dt22tAz3lJShqqa2ohxyqtqsK30hGEZjpysxIHj5YbjO0l5VQ22HzzptRiGcXUOgYjyAPQFsBhAayHEPvnUfgCafouJaAIRFRFRUWlpqStyMsZQPvXBE7/DmOfma8YZPPE7jJ40Lyhs5LM/YNDE7wK/C1ftw+jn5mPG6n2hlxtC3WP517cbcfGkedgaUoH0f3I2Jry7FK8v2I4N+4/j4knz8MKczZbSM8uU+dtwyaR5WLHrqK33vWRS8HP0A1+s2IvxU37CwL9/h4snzcNzszfpxt11+BRGPjsPf5+xIeI9f//RSgx/5gecrKg2JEPfJ2ZhwFNzTMntFPdPW45h/5qL8qoazfPK/Jdf5kVcUwhE1BjAJwAeEEIENQeF1GzTfCRCiClCiHwhRH52drYLkjJW2BOhpbr3WHBrTXFXrbBRbj3H3IoWwNIdRwAAJceD0zheLlUmq3Yfw76jkjx2V9B6rNotpbPniL2t+QMhefQD60N6est36j/jwycrAQBFOw5HvOeiLQcBABXVUk8insxOF2yWZK92qTcaK64oBCKqB0kZvC+E+FQOPkBEOfL5HAAlbsjC+BCPPnCeC3EAE6/SaNQU+UXVxpEiiFfcsDIiAK8DWC+EeFZ16ksAN8jHNwD4wmlZGHtwqh51y3xUuDwtmcz1mBGlGy0KhSiERHqcfmuUuLGF5mAAvwWwmohWyGF/ATARwIdEdAuAHQCuckEWJgEhnWMz17mB3z7+eCHw3BJJE/gUxxWCEGIB9L+9EU6nzyQPbrf8mXCceAOh+iARe1x+yRKvVGY8J9aPwWzL2+0KJRErMD2cmPBN5DkEv3UaWSEwuhw6UYHHp69DtY6duN0fv1Kxf7Z8N977aQeemrEetRrWGVU1tXhs+tqAlYqmbDpqRqCugiaVJtl/rBxPFq4ztDZh1+FTmPj1hqj5n75yb0QX4St3HcVr87YFhSl5O3QiNguiNxduR1FxuPXOmj3H8MrcrSivqsGfPl4ZZp5rhadnbkBeQSGOna7C9JXGTYeNlh/lNSmvRv1u1+w5Zugeby7cjqUqa6apP+/EQtl6ianDjTkEJk555Mu1+GrVPvTv1Byje+UEwsnmwfDQeuF3H9S5NR7ZozX6d8oKOj9j9T68ubAYx05X4dmr+tgiwx8/Xon5mw9iePfWGNi5RcS4d76/FGv2HMe4vm3RvU1T3Xj3Tl0OABh9VhvN81e8tBAAcNsFZwTCZq07gDcXFuPwyUo8f3Vfs9kI8Nj0dQCA4oljg8Ive3EBAKBXu6b4sGg3TlbW4KVr+1lOBwBenrsVADBp1ibsd2BBWGDISEOBXPbigrA8ahH6PB78dHXQb6/xiykt9xAYXaprZKuOkLLqZuHVGiZQgkJb8+qoehZLerIrq2WNzENUVWs/FztQ8qQ8e6dQ0jl+WnuFuRWcGtJRGiDK7X1Sd9qC3Y2rWGGFwETFy+9P6+MPrbS1Kv9IFbtyxuqnqHzDiVAx2VkhaT0PQ/ePEiclRf/+jL2wQmBM41SrRuuukSr2WKVQZ8PJykbJg5/agk5k17EeAhJ3UtlvsEJgoqJXkdn1eUYcprF2ytZrEhk7lZRZ7wxGowd6ZOZu7yuiDbP6JW+sEJioeDpkZDKukc6L1sdppdMT1+seHBFd47nacNeUwByCc3M3jAQrBMYybgyBRBomUNI3VZkH3c5aDkInOY1fZ08cOwgMY9mYnlMVtSJinPiH0yReRGeFkKD0fvQbXPzsD0FhI56Zi3Me+9a2NEIL+aRZm5BXUIjyqhrU1grkFRSaup9W5fTb13+2LN91/1msKYPWx6lVmY1+bh7OengmAMmvfV5BIfIKCm1RhJ8u220oXpe/zMAV/15gOZ1Ln5+PvIJCrN4dbK9/81tFAMJV4hNfrTP03k5XSs/j4S/WBMKmLdkVFu+HTaW4/MU6+a969UfkFRRiwFOzA2HRfRlJ/1fuOoq8gkIs2hq8fiCvoDDg5fb7jeZ8ZHZ6sBDjZPNfAFi87RDyCgqxfKd0v0MnKpBXUIjHp0vPZeaa6Osshj8zF30eD/7Ozn70W1z16o8ApP0a8goK8dj0tej3xCwAUvl79Mu1pr8Zu2GFkKCUlVdjc0nwoqOtpSdxzISZoV7rUe8DfvvHYgBSZVFjornoSMtSAAs0Fh6ph3m08qe2WNqwvwwnKyU/9naaZwLA5yv2GopXXSuwcrexxVdarNsnuaNevP2QofivL9huKJ5Sjt75cUfUuKtVi8d+3i4tDjPjulvpkf24TcqDVqWvLDL7du0Bw/cFpLKndoM+d5O058qirVJailv3NxZKz6Vwtf5CQ4VtpSdx9FR4eVHyrtzzzYXFQeffWlQMr2GFwOgSS0XthUWNEMY8pkbKl1vzAn6xONKzGIs2CWrHUJPZ8hVpHYJzw1UU8juxYYXA2IZXk31WKye31iFo2+dbTNwiVitfPdwUPzQtN4tZ6HuyI22td88rlRnf43alZXR9QyzfjnRtpInq6DJYfy6kceRP/FE9SdSZnfpJqsSEFQKjS7SK165GjdXbRFIgRu7ppdcAv7ksCCX6kJF98ke7VUBJG3JdYa/SSCEeMmKYmHDNdNKCRrKzZx5LizXaI7K7Nax3Pz05opl4eqvP3OspOJFPbVcr/oC9nSYQ20pP4OCJyjDvoFaJ9jGEWiypK+jQAr6k+DDmrC9Bv9zMoPDqmlp8tmwPAMld9E2D87TTOlWFRVsPYkzvHM3zgDGTw5lr9yM1VcrYNyqLlN1HTke+kNSH0o+fth1CyfEKNG9UL3CusroWT81Yj6v7dwiEhVqcVNfUYs6GcFlnrN6HHYdOaSY/Y/U+DOnaEmXl1dheehJDuraMLG8I01fuw4QLOodnS+cdK+s/9h49jbd/LEb/vCxsPFCGiqpajOjRCsdPV5tKf+Wuo7qutoUAPlm6G1U1taiorsX8zaV44Zq+aJiehpMV1dh4oAwAsO9YeSB+KAu3HETHFg1RWhZuvaR2e22W0Ofz5cq9eG58H6SkSCdW7DqKWiGw58hp1AqBUSGebRUrNYVlO49g8bZwecrK657nzkOnkNuioWWZY4EVQgIx/Blp3YHdLn3taFX/z+QfNcPfWlQccJm8YX8Zrn1tsWa8e6ctx7xNpVjw52GBsLphC+n/rHUHMCiK62oAKFwVbkuumAIaQUn2qRkbws69MGcz3lpUHGRC+HPIvgR/K1wfdt2WkhO46/1lgd+nQyqSu95fhot7tsaiLQdxsrLG9DtevecY3lpozKQUAGau2Y9xfdth1HPzUFZejVd/qNu34fk5m02lDdS5+g5GKlgrdh0NMv0EgP/7fC2eueocPPR53ToHxexUi5+LD4c9Z4VfvaJd9oyg1ZqfumQnfjOgIwAErWEAgOsHdox4v1++vChqmhf883vP3HLzkBHjKaUhG8FsLdFuRe4+LLWcy6u0N+vxCyVl0fcD2H0kvBdwqjK4xa01ZLPnyOmwFqcZ9mva/mt3EZTen7rl6ib7j0sKeo9Gz81NgxytHtThE/obM+0/Zv9+EG7CCoGJilnTOwLZ/tHWyeCX0VbzRDJX9YnVoW9QWuZacx9uWhuZnULwua1AVFghMFHxQ2Vl1n+QkzLH+TcfhO5q9ETKZAxoWVP54HNwDFYIjKeEjtFG+9i0zrteeTmQYGi+NBcv2Z6qf/FLb8rsqzayjsXPsEJgomL0oxBBx/Z+tXX76hqrEPzcwjXyZNyq9Lx8TFbz6OpKZRfT8gOsEJioeLmnsoLZ1apeDXNZruRCLtR2vGc/flWcflm45xc53ILNThOc4c/MxcOX9cRF3Vq5mu7ny/dEjdPrkW9woiLYikVL2Yx89oegcOUb/Wz5HvzinLZBcRUvlde9Xme+Gsl9dKi74X3HTuO8J2dj8nX9dK9ZGWIiGStXvrwwzLpq7sbSMNkUz6UA8Pj0dYFjoy6TJ/+wNSzsm7UHkFdQiAV/Hob2zets33ccOoX+T84Oi+8WgR6hxrmPlxpzHf787M24Z3gXw2ne8tYSfCevZfnnNxux79hp/KSxZkBZN/H7j1ZGvF+P/5tpOO1QTlfWICM9FQBw39TlyGvZCP978ZmW72cU7iEkONtKT+LhL9a6k5jq6/3zJ6ujRg9VBnpsKTkRMIMMHTL662ero7aczbiP/nTZHpSWVeDGN5cEhRsdG7bSoFy+8yiOmzTvfMPEmgIjhCrw/y7eiRKNRV5uUTeHYL2rN2n2JlPu3udsKAkqW+/9tBNbdMyg9ZSB+v2frrJuIry5pCxw/OXKvXjBwtoPK7BCSAIsewO1uB2lE2aBVncpc5tI8vl58CF0aMTrkRK7kndieDNS+bbruXk1Oc0KIQmwWrT0viUv6uQUnTkEn+sHTfzi6jgSblRHhibXHZfCn3ilkFkhMDFht1mg3qWBhUoeWd+w62V3satH6MhmfBF7gfbU5KFeVt3CcYVARG8QUQkRrVGFPUpEe4hohfx3qdNyJDNWLSUMDRlZurN59GQRwuaN4h3MUTxZrHgta/w8KWdI8aip7kaybwEYrRE+SQjRR/6b4YIcSYvbH5cTVap6HUJwWja7iZZvF1YhcgfBVezSR3EwOqdJws4hCCHmAbDuf5YxRGW1vtM3o9YiQggs3XE4zEFbTchXVav6ffhknaMvJfTAcesOvvQ+4L2y07DSE+UoUrkzPnyyMkiGWFHMVtXWKVtKyrArmntsma80PKmq2XX4lC8d9P207RCOqJ6jUQuwWDimsRG9wtbSkwBi18OhTgPX7DmGnTouxo1yvNy45ZJVSssqsGjrwTD5ncbLdQj3ENH1AIoA/F4IcUQrEhFNADABAHJzc10UL7548FN9M0+jH/cHS3ahQL5P8cSxAZO7F+ZsDrL3f/n7Onv28zRs1cc8P99Qela4+a2ioN9VNQIzVu93LD0AGPnsPMNxo5kaDn36+1jFcYT5mw+i7xOzXE3z1neKdM9tP3hS2scgxib+hf+cG/T7she116Ro7aOgx5sLi/VP2tSwV9bR9Mhpas8NDeLVpPIrADoD6ANgH4Bn9CIKIaYIIfKFEPnZ2dluyRd3LNhSGvM91AufgLoNSUJtsZfv1NTdDGMrepsFOUGZC61+K6wP+SadxhOFIIQ4IISoEULUAngNQH8v5GCC8cqyQQ1b8zAKRG4aLdhT9r3/gmLDE4VAROp9EK8EsEYvLmMMOybPUlOMFWe9pOLBvt5L+PmYhx+Zuzg+h0BEUwFcBKAlEe0G8AiAi4ioD6S6pRjA7U7LwUTHqEJgmESDS76E4wpBCHGNRvDrTqebbNjR4/XBiBG3CJkAbppe+qHs+wFeqcwESOWvwlFY15kn3uaUvF7QFyvs/jqOeWXuVgzrno3crIY4oNpA/devLAqL+9q8bWFhNbUCT8/cgE4tG6FxgzS8PLfOnPTJwnVBG6wfL6/C87M3Y+zZOVhlwnsoU8cnBt02M3Ws2eOOlc0ymyznpq/ca8t9vIIVQhzzj5kbMGnWJtw9LNjne9GO8ML95Iz1YWHzNpXiVQ1FAQCvzQ92rzxp1ia8ubAYry+w1+1yMvHtugNeixBXuNnY/t0Hkfc2SBZ4yChOUSxWKmtqUV1rbeVrTa3x7nh1TfS48dW5ZxgmFFYIcUqiTr4maLYYJi5ghRCn1CaqRmAYxjNYIcQpijogYhtqhmHsgRVCnGJHB8GXFnLc8WEYz2CFEKco9tl+rNMZxg7i3aY/HmGz0zgg/2+z0aJROr753QWBsG4PzQQA1Arghe+2mL5nXkEhcrMaGo5v5Ns8VRnZ9bMRKmv8t1cA4w33TV3utQhJByuEOODgiQocPGHcX7tRdh427l6Y57AZJvHhISOGYZg4wA1vuawQGEPwcC7DeIuZhaRWYYXAMAwTB4Tube4ErBAYhmHiAIseakzBCoFhGCYOsOqzzAysEHzGrHUHcPRUpea5o6cqMdsGj5mPfrnW9DUzVu+POV2GYazDPYQko6SsHLe9U4Q731umef72d5fi1neKcChGE9S3FhWbvsYJs1eGYYzDPYQko7JaeuE7Dp3UPF8sh1cZcEXNMExiwZPKSUa0pfq8OIxhkhceMkpSotX7vCaAYZIPHjJKMrieZxhGD+4hMAzDMAC4h8AwDMPIuLFLIisEH6L33kvKJNPP1xdsd1EahmH8QDX7MkoujE4WT5m3zVlBGIbxHTyHkKQI3keSYZgQ3KgXWCH4CGI7I4ZhdHBjHZJhhUBEQ4joJvk4m4g6OSdWcsML0BiG8QJDCoGIHgHwZwAPykH1ALxn8No3iKiEiNaowrKIaBYRbZb/NzcrOMMwTDLhpx7ClQB+AeAkAAgh9gJoYvDatwCMDgkrADBHCNEVwBz5N8MwDKODn+YQKoW0oacAACJqZDQBIcQ8AIdDgq8A8LZ8/DaAcUbvl0gs33kEB09U4NipKhw6URGwMiopq8DW0hPeCscwjK9wweoUaQbjfUhErwLIJKLbANwM4LUY0m0thNgnH+8H0FovIhFNADABAHJzc2NI0l+UV9XgypcXBYUt/suIwPGIZ37A9HuGoHf7Zm6LxjCMDxF+WZgmhPgXgI8BfAKgG4CHhRAv2iGAuuehc36KECJfCJGfnZ1tR5K+oKom3Kg41MZo5+FTrhQChmH8jxs1QdQeAhGlApgthBgGYJZN6R4gohwhxD4iygFQYtN944bUFDYxZRjGOL6YVBZC1ACoJSI7xy6+BHCDfHwDgC9svHdCwR0EhmEknK8MjM4hnACwmohmQbY0AgAhxH3RLiSiqQAuAtCSiHYDeATAREjzErcA2AHgKpNyMwzDJBVuNA6NKoRP5T/TCCGu0Tk1Qic8KdB6uVrvmzsIDMMAPplDAAAhxNtElA7gTDlooxCiyjmxGIZhGDW1LtidGlIIRHQRpPUCxZCMYToQ0Q3yGgPGAlqv9vrXfw76/WHRLhw5VemOQAzD+Brf9BAAPAPgEiHERgAgojMBTAVwrlOCJSMbD5QF/f5hUyl+2FTqkTQMw/gJX1gZydRTlAEACCE2QfJnxFiE1xcwjDaPXt7TaxF8iRuuK4z2EIqI6D+oc2j3GwBFzojEMAzDhOEjK6M7AdwNQDEznQ/gZUckShK4f8AwjBn8NIeQBuB5IcSzQGD1cn3HpEoCeMSIYRgz+GkOYQ6ADNXvDACz7ReHYRiG0aLWL87tADQQQgT8McvHDZ0RKUngHgLDMCZwo8owqhBOElE/5QcR5QM47YxIicfcjSXIKyhEXkEhDp/kdQUMw5jHDctEo3MIDwD4iIj2yr9zAIx3RqTE46tV+wLHOw+fQlajdFdMyBiGSRw6Zzd2PI2IPQQiOo+I2gghlgDoDuADAFUAZgLY7rh0CQivP2CYyPAXEs6Ng/LQIcv5UfpoQ0avAlDGOAYC+AuAlwAcATDFQbkSFqWws15gGMZvRBsyShVCKPshjwcwRQjxCYBPiGiFs6IlNqwPGIbxG9F6CKlEpCiNEQC+U50zOv/AqOCeAcMwfiVapT4VwA9EdBCSVdF8ACCiLgCOOSxbgiJpBJ5LYBhteHPZcMilhxJRIQghniSiOZCsir4VdbVYCoB7nRYuEdl04ATO7ZiFuRvZiynDaMFNJe8wsqfyT0KIz4QQ6q0zNwkhljkrWmLy4KercfRUJX7/0UqvRWFcpGF6qtciOM6wbtlei8DEiNGFaYyNVLuw8xEj8at+7R1Po3H96NNpg7u0DAvL79jcCXE8oXjiWLx5U/+IcZo3tMdjfvHEsbhnWBdb7qXFzYM7OXZvv8MKwQNS3BoQZHyD1pRRshUDO5tBNTwH5wisEDwgJckqgkTH6usknj61jJOO3pJNUathhcAwrqDVRXBfCi+xsw53Y8P5ZIQVggdwb9c9/OIziusve3HyefpRT7vVm2SF4AFcNzCAPyseJ7Fz7Y0bewMkI6wQHOL6N37GFyv2YPOBMny8dHfQuX5PzPJIKsYreCEi0KSBPVZGAFA/zTkzXp5DYGxn3qZS3D9tBZ6fs9lrUeKSq/KdNxe1wk2D89CkQRruvKizqeu01IHRimfydefi3Vv6o3N2I904r9+Qb0oeNzkvrzn+OKobPrlzEK4f2DHs/NCuwSa5at05afw5GNG9FUad1RoA0DOnKQDgvhH2mJ2+cE1f09f8aXQ309d0aqn/7ozglpJiheAw3C60xkOX9bT9nm2aNoj5HhecmY3Vj47CAyO7Gopvh7386F5tMLRrNn538Zm6cUb0aB1zOk7Rq10z3D2sC9o0a4DHr+gVdv5v48LDFK7s2x6v33ge7rhQUsD10qQqq2G6Pa7U+udlmb7m6vNyTV/TrXUT09d4ASsEp2GNYAknGkR2mvuaneTjEaPYUOYMUl1oKVMSjxmxQnAYnvyyhhMfpS33NPk6FSVkRylI5nULNbXS/1SbF/FYKRJWJPCLtVs0PHVhTUTFAMoA1ACoFkL4dyDUIqwP/IOdOiboXpHuK0e0Y1I5XioVJ6iR7UzdWOWfvGrXH3saDBNCHPRaCKdI5o84FpwZMnLoU+dXHBE7ejZKT9sPbl+s9Spik9utXPOQkUn2HTuNiuoaAMCOQyejtvy2lp6MeJ5xDztGG8wqeCVJO3qKyT1kJD3ANDcmERwgXhqGXisEAeBbIlpKRBO0IhDRBCIqIqKi0lJv9xCoqRUY+Pfv8MC0FVi9+xgu/OdcvLWoOOI1W0pOuCNcgkEE9M3NtPmesVcmHZpLG50H3SnCbc9u3wyAZGkTSr1Uc59fXsvgTdbP9aG31GYZ5tcahCrL7jnhFjltMzMAaHuNtZtom9knsmL2WiEMEUL0AzAGwN1EdEFoBCHEFCFEvhAiPzvbW3/rSrf123UHUHxIavkX7TjipUi2EaudtFmu6R/ZdI9AmHrb+bamqaUP1jw2Crk6FYA6/ne/vxALC4ajqwnzwZE9WmFEj9ZYWDAcI3u0Cjp397DOphXCWW3rlEpuVkO8f+uAoPPL/+9iU/eLlVFntcbGv43Gkr+ODIQtLBiOFQ/HJsegzuGVfpdWjbGwYDhuv+CMmO4dilbV3rVVY0vrE9Q8cnmw2XQkJbKwYHhMadmJpwpBCLFH/l8C4DMAkR2qe0zitgvcX52ZZmD8pkE9e1ejao0/N66fhpxm2usTMlTpn5HdGO3kVipgrLeh7JOgvk6hVZPY1kRkNUoPez7NG6XHdE8jqK18+nRojvppqchuUj8Q1rh+GjIbBssR7VEZLXvtMjNssz5r2bh+xPMdmoe/swAGRAjdIyPSkJFW+fAKzxQCETUioibKMYBLAKzxSh4jKK+U3RDETjR94ISC0ksz5repcwN15eWDuVDf4sfPyYciuYKXVkatAXwmfzRpAP4rhJjpoTyMi3ix+MexsV8PKnuvFEzi6DW5ytdrJETQCEaevd3WUG69b88UghBiG4BzvErfCkohSeaVjHbhxSO0dR2Cyfh2t4L90Krmz0CfFK9nZy0Sp2J7SyIOGbn9bUdrQTkzZOTUmJF5hBBx2dpWP0K7PgN/Khb9zBkRN14tkVghRGDmmv2Yu7Ek8FuZGKoVkjdTAChctQ/Pzw72aBqPCsNtib3YRtSxlcp6cRxK24n7OUk0Ub38XLQqbjtGAOLp/ahhhRCBO95bihvfXKJ57iPVHgeTZm8KOheH+iCIp67sHTXOr/pZd0/dM6cpbhzcKWKcWFpYdng1Vfj1ue3xj19Ffx7RUJeJnjlNcfk5bW27XyT0rGm0vLXefmGwSaeWjFOuN+ddRnrXeaaucYKxvXOCfr9wdV+cf0YWslSWWU9d2Rt9OmTi7PbNIj5fAUR0RQ4Evx8j31OGzRZ1VmGFYAI/VvSxViwK6uo31D+9FpFcFis0Stcu5DPuHxrV1M5MC6tBvbpi/Mmdg/D+bSr7fAPvLNQkcO1jo6RwAfzrf87BeA13x7G0ImfcPxQtopg92kXRQyODfhdPHIviiWPxwMhwV9r3jwhWEi9E7BmfAAAXJUlEQVRq2OIP69YqsBbAyCOYcf9QtG8eeaGX0+R3bI6XftMPxRPHBsIGdWmJaRMGBpnRXjsgF5/fPdiQuXO0dTTKmqVLe7fBtQOiu8tW9nsIJbOhtNDPrXlLVggO4Kbe8GLoxc8Q6Q9ROPZN6b1wVXrxMIQQr+PeThDpGzbSMIxXAxRWCDahnjdwcw7BCWdffijDZkQIrcjMfoR6ryvW56CWy4+9y1D88N6dwMqjj/a+opUx5fJYv0+3yw0rBBNEejmK8y23ceIjNlKh+tVZFyGRbOUZX2KohxC8oU+0it0vXxMrBJuoUfcQXEzXq26+0/rPTCs/NKqudalJmeOhVZ8I+PExR+rlG2kMKZenxNmYLisEE0QqCOoegpsViV3lzfwwi7OZdPMz8qpCStQhGr9hd1k1cru6LT+llxz78KM7+GGDHNf5xb8X4ER5Nb77w0UAgLyCQgBAj5ym+Pr+oWHx1+w5hsteXBDxnj0f/gZpKYTqWoHx+R1sl1kPuyoVsx9NmoGlmF1bN8GKXUetimQZvV5Tp5aNsHbv8bDwvBaNsFTDa63RZ9u5VWPNfLaL5CDNJojsa4CoLW7ObN3YnpsawOnKztIcQoRzaamEVk0iW4kpVmSKhVW89DaTUiGs2n1MM3z9vvDKAgCmr9xr6L7Vci/hg6Jd1gSzgF7l16BeCsqrag3dY+YDQ3H3+8tMpZuhYVL6p9Hd0Et20ZyaQjirbVP0eXyW7j2+/8NFGPavuZrnlMp4aNeWmL/5IPp0yMS4Pm3x6PR1aNk4HR/dMQiv/rAV05bsClkApv08Ljs7B//41dn4atU+AMD0e4agaYZU/P82rhfG9GqDW98pCrrG6Ec8+bpzsWznEaSlECa8uxRpKYR/X9sPI0JcXpvl07sGBVqYoSiiLfzzcOw/Xm7p/t//4SKcrKhGepqk3OulpuC/tw7A0dNVGNApy9I9FabfM8Tw3gi27DcdQTGqn+HX9w817Xb807sG4eipSpQcr0B5VQ2aNKiHy87Owb1TlwOQ1lpMGt8Ho56bF7hmZI9WmHzduWFuz8f1aYvPVxirT7wgKRWCWWp9rN71WrF9OmTip22HNc+lp6WgsrpOWXRv09TQkJHSA1Lo1LIRth+s2xHurou6GJS67no9FHn652Vh/uaDGNylBfrmShvC5DTLQKeWjfC/l5yJaUvCla9WVoZ1a4VGKpfEvdvX7S2QkZ6KkT3r7MCN9gwUpdssox4u7Z2D8qqawPWje7WJKlM0+uVG3wCnbWZGYPOYaISu/dB6/oNs2oBG/XzdoGXj+igtq9A8px7H75HT1ND91J+81ntQfy99czPRrU2TsPPqMqBEH92rja8VAs8hGKDGWEPbE/Qq8mSyKbfb1tuo/g+NF0kMu9sUyfN2jRHpeViZZ7Pbii6qlZHOebfd4LBCMICfewh6hT3RJizVCi40b6EfjV15t3ofN5Sxf0ukeZx+Wqm+svSJPASoG+5SFlghGMDPzuosLXzxb3bCMCJq0EY0dqbNZqqu4PRjc/obSaTXzgrBAB6tOTOEXlmPpZVsVAG62e4ykx+1grDy6sw+u7D4hq+3/gT91Ob1O/7qIfibpJpUXrnrKIpCzAtDK79xLy1EdpP6aK8yGVxSrD056wd0/faYrDKMxPabXtR2XRz7feOhpR8HIrpKpPdupYeQrM83qXoIV7y0EE98tS4obN7mg0G/V+w6ilnrDuDNhcWBsA37y9wQzxJ6E6qXhrj7VXPvcHPWQHZy46C8oN8XnJkdMb66cu7YQrLp/u35HSNek9VQf7P54d1bBTxIxoryHBUzRuX/vcOMPd+r8q27EI/EjYPyMK5P27D5pVuHRnY5bhQ7K8sUQpBNf0/ZCujKvu1M3ee2oXWuuztkZSC/Y3M0aSC1d8efZ35dUFd5HUbTBva0mf9HXpvUu30z9Ncw6f1lP3P5dYqk6iFocfiktqlaPHB2iGnfTw+OwPl/n4PWTesHCl3j+mk4UVGNVk3qo0Q2y7t3RFc8M0vaw6F9LIunDDS81C6HtXjn5v6BY2WBoN41mQ3To94PkMxIlXi/+2BF0Lk3bjwv6vVGG5T3DO+Ke4bXuYxOTSFD8ilcclYbFE8cG8i3ET69axB++fIi3fPq9J+7Oth99U1R9qBwG7Wsoc9t0vg+mDS+j+F73Tr0DPytcD0AYP6fhscsW6smDUy9SwAR3+XFPVsH7vfh7QPx+PR1eGPh9sD5Yd1amS4LTpBUPQQt4mF4wChKRaae8/DzhHisKKaBPELsLvy8XURxo+3SU2eF4JP60srYt5G5TCV79oyte/ewjIpvx4fjlzKhBVfG/sPP5cUsrBC8FiBGtCp6dQE1XJHGYU2TTIvvFPxQXv0gQ7Lg+l7nLqfnO/yy6MyOqq1ugtl8nnzyGMLwYt+FeFSODGMHSa8Q/NLcsep+QWsFb6zrJnypHCI9nySqwP2QVT/IwDhDUlgZLdpyENf+Z3FYuNcz+mqaNEjD0VNVMd1D60NVbLAbpqcBCLeoaih7LVXXt3qbeoTqiUbp3hYfRcxG6WkoK68GEK43GtST2jxpqcarMUXJNqoffbN1oyTS4ig/the8pH6a+Xa10Z5vo/qpOKHySOs0SaEQtJSBn+jSqjEmX9cPI5+dFxSenpqC7Cb1sefo6aj3OEdlgipN/koFLrtpfdyd3wVje+dg6NPfh133+g3BZpjj8zugbbMGYR5RFb66d0jAffirvz0Xny3fg3PaZ+J4eWzKDACeurJ3mCltJFo0ro8Hx3THmF45uOCf4XkDgIIxPZDVKB1jI6zLCCUjPRUPje2B4d1jc1+tpk+HTLTLzMBjvzgr7NxbN52H8qoa3PGeORfkbvDBhPOx/3g52jdvGOTZFrDeU/j6/qFhC0RjQdnD5K2bzkOFRpmNxts398fpyhrT171xYz6eLFyPP4zqFggzW4ZDeeU3/fDJst24b0RXzFyzH3dc1BmT527FnRd2tnxPMySFQvA7d17YGZ1aBm9IktmwHlY8fAmACD0ZokCL+PJz2gZ6A+q2BwG4I0Jh6pDVMOj3TUPyQES488LOeH7O5rD4vdo1Q692UoFvm5mBuw0uwjLCtQNyw8KiDV/dLuetR05Tzf0smmXUwx9HdTcty62qhU52QERYWKBtH39Rt1ao9al/lAFntAgcn9sx2A20VYl75DQ17Iba6P0A6Tla4cIoiyP1GN69NYZ3bx0UplWGzTCmdw7GyI2Xs9tnAgD+NNp8+bUKzyH4hFgHFIRQzSGot/M0LUe4UmEYNYkz+MWEwgrBpxi3u1cf22CDz6ogPvBw5p9LSOz40nADHisEIhpNRBuJaAsRFXgpi98wvek9tHsFVhd0+aUVGFhYZzA+m4y6Bz/qxMMzhUBEqQBeAjAGQE8A1xBRT6/k8RrLm7GoriPlbapdVxi8j5GegV9bNUD8u+gw/f5Z8zEO4GUPoT+ALUKIbUKISgDTAFzhoTy+wsjnro4jRN1vvyy284KkWb3sg3fsvQSM3XipENoBUO+QvlsOs53O2fqbufsBrcae2uTxrLb6FhmKJUK3Nk0Ctsoje7ZGpuwCWs+CItQ07oKuUrxIrqMv6dla95xTKHmP9AyAOjfaLRrry5/MtMuMwautT+ndzrp5p9f06ZDptQia+N7slIgmAJgAALm51ky6xvVpF3D37Bb/vW0AzsppBhCwpaQMFdW1uPa14PUQI7q3wpwNJQCC5wwWFQxHy8Z1PuI/vmMQig9JNuBjnp8fdI/Lz2mLc9pnIlfeK+DHB4ejRaP6SE9LwcKC4Wit8jWv5oMJA1FWUbd2oGBMd9w4OA+tmjYIinfPsC64ZkAuUonQvJE9+wiYYdRZbTDvj8MC+dPjT6O64fqBHdE6RH4zrHj4Yl/0MCLJofT+zCx0K3poJDLq2bfITsHrJ/Xh7cFlOJ4Y17cdcls0RLfWTbwWJQgvFcIeAOqdK9rLYUEIIaYAmAIA+fn5lnqpeitvnWRQ55aB43M7ZqGiOnzhS2OdzTfahrTmMtJTI9ptqyvLnGZ110ZqFWakpyIjva6SSEtNQfvm4ZVuSgp53rqMpgwAffnNkBmhd+QmkeSokdddmVEI6sZFIhFahuONfrnNo0dyGS+HjJYA6EpEnYgoHcDVAL50IiFLm2zbjOZ2j8p/GyaU7cYHjyypMGpVViOvMbHq+8pOeA4h8fCshyCEqCaiewB8AyAVwBtCiLVOpOUHNzJ+kMEMPpizZDRQhozS4q1AMXGBp3MIQogZAGY4nY4fHIv5oZfCxD9KD8EPZdp7CRi74ZXKLqG5kY37YhiG9Zc/qZF7CNzAYJwgKRTCR0W7vRYh4pivVcsWrhKSj1of9RCYxCMpFMLGA2WupjdQ5SFSzeAuLfDva/sGft87vAvObN0Ywyx6aTTbw3hiXC9c0aetobhXn5eLLq0a4+rzOkSPzNjCwDNaYPJ150aMM+CMFujepgn+qHK57DbXD+yILq0a48q+jiwbYjyE4mnJf35+vigqKjJ9ndWNcP59bV+8++MOLN5+OCj81iGd8J8F24PCJl/XD6N7GfO5r8hTPHGsoXCtOADQNzcTn9012FCaDMMkL0S0VAiRHy1eUvQQrJKi2m/Aj/hYNIZh4hBWCBEgaI/vx0+fimEYxjhJoRCsL/zS7iHE0SgbwzCMYZJCIVg10UshNr9kGCZ5SBKFYPU6MmES6r7m8IP7AoZhEoekUAhnqjwKZjepb9hZG5G2YzWtzWSaZbjvCTQ3KzZnbgzDMGp87/7aDt69ZQDW7j2GtJQUaW8EAqb9vAuZDeuhR05TZNRLxWUvLgi7LoUID1/WE/XTUnDHhZ1RWV2LkrIKfLVqb1C8ydf1w8DO2msP7GbeH4fh0MkKHDxRicFd3EmTYZjkICkUQlajdAztGrxRzH0juka9jghoUC8Vj1x+ViCsQ1bDMIVgdP2BHeS2aGjIHTTDMIxZkmLIyCp6Y/RsZcQwTCLCCiEC7C6GYZhkghVCBNijJMMwyQQrhAiwPmAYJplghRABvTUI8eQQkGEYxiisEGR+e37HsDC9OQS1VVFDC5t8n5HdCN1UayPU3DKkk+n7MQzD2EFSuL82g9q99Ed3DMR5eVmOpscwDOM07P7aBngKgWGYZIIVQgTYVxDDMMkEK4QI8DoEhmGSCVYIEeB1CAzDJBOsECLA+oBhmGSCFUIEUnnMiGGYJCIpvJ2a4ev7h+LTZbvRMD0NPXOaei0OwzCMa7BCCKFHTlP8dWxPr8VgGIZxHR4yYhiGYQCwQmAYhmFkPFEIRPQoEe0hohXy36VeyMEwDMPU4eUcwiQhxL88TJ9hGIZRwUNGDMMwDABvFcI9RLSKiN4gouYeysEwDMPAQYVARLOJaI3G3xUAXgHQGUAfAPsAPBPhPhOIqIiIikpLS50Sl2EYJunxfD8EIsoD8JUQole0uG7sh8AwDJNoGN0PwZNJZSLKEULsk39eCWCNkeuWLl16kIh2WEy2JYCDFq/1G5wX/5Eo+QA4L34llryEbwmpgSc9BCJ6F9JwkQBQDOB2lYJwKs0iIxoyHuC8+I9EyQfAefErbuTFkx6CEOK3XqTLMAzD6MNmpwzDMAyA5FIIU7wWwEY4L/4jUfIBcF78iuN58dzKiGEYhvEHydRDYBiGYSKQFAqBiEYT0UYi2kJEBV7LEw0iKiai1bLjvyI5LIuIZhHRZvl/czmciOgFOW+riKifx7K/QUQlRLRGFWZadiK6QY6/mYhu8FFedB0zEtGDcl42EtEoVbin5Y+IOhDR90S0jojWEtH9cnjcvZcIeYnH99KAiH4mopVyXh6TwzsR0WJZrg+IKF0Ory//3iKfz4uWR9MIIRL6D0AqgK0AzgCQDmAlgJ5eyxVF5mIALUPCngZQIB8XAPiHfHwpgK8BEIDzASz2WPYLAPQDsMaq7ACyAGyT/zeXj5v7JC+PAviDRtyectmqD6CTXOZS/VD+AOQA6CcfNwGwSZY37t5LhLzE43shAI3l43oAFsvP+0MAV8vhkwHcKR/fBWCyfHw1gA8i5dGKTMnQQ+gPYIsQYpsQohLANABXeCyTFa4A8LZ8/DaAcarwd4TETwAyiSjHCwEBQAgxD8DhkGCzso8CMEsIcVgIcQTALACjnZc+GJ286HEFgGlCiAohxHYAWyCVPc/LnxBinxBimXxcBmA9gHaIw/cSIS96+Pm9CCHECflnPflPABgO4GM5PPS9KO/rYwAjiIign0fTJINCaAdgl+r3bkQuQH5AAPiWiJYS0QQ5rLWoW7y3H0Br+Tge8mdWdr/nScsxY1zkRR5m6AupNRrX7yUkL0AcvhciSiWiFQBKICnYrQCOCiGqNeQKyCyfPwagBWzMSzIohHhkiBCiH4AxAO4mogvUJ4XUT4xL87B4ll3GsGNGv0FEjQF8AuABIcRx9bl4ey8aeYnL9yKEqBFC9AHQHlKrvruX8iSDQtgDoIPqd3s5zLcIIfbI/0sAfAapoBxQhoLk/yVy9HjIn1nZfZsnIcQB+SOuBfAa6rrmvs4LEdWDVIG+L4T4VA6Oy/eilZd4fS8KQoijAL4HMBDSEJ3iRUItV0Bm+XwzAIdgY16SQSEsAdBVnrlPhzQZ86XHMulCRI2IqIlyDOASSM7/vgSgWHXcAOAL+fhLANfLliHnAzgmHPYLZQGzsn8D4BIiai53/S+RwzwnZH5G7ZjxSwBXy5YgnQB0BfAzfFD+5HHm1wGsF0I8qzoVd+9FLy9x+l6yiShTPs4AcDGkOZHvAfxajhb6XpT39WsA38k9O708msfNWXWv/iBZTWyCND73V6/liSLrGZAsBlYCWKvIC2mscA6AzQBmA8gSdZYKL8l5Ww0g32P5p0LqsldBGsu8xYrsAG6GNDm2BcBNPsrLu7Ksq+QPMUcV/69yXjYCGOOX8gdgCKThoFUAVsh/l8bje4mQl3h8L2cDWC7LvAbAw3L4GZAq9C0APgJQXw5vIP/eIp8/I1oezf7xSmWGYRgGQHIMGTEMwzAGYIXAMAzDAGCFwDAMw8iwQmAYhmEAsEJgGIZhZFghMEkBEdWoPGGuiObdkojuIKLrbUi3mIhaWrhuFBE9RpJH0q9jlYNhjODJnsoM4wGnheQiwBBCiMlOCmOAoZAWKA0FsMBjWZgkgXsITFIjt+CfJmn/iZ+JqIsc/igR/UE+vo8k//uriGiaHJZFRJ/LYT8R0dlyeAsi+lb2b/8fSIu8lLSuk9NYQUSvElGqhjzjZWdn9wF4DpIbhpuIyLer65nEgRUCkyxkhAwZjVedOyaE6A3g35Aq4VAKAPQVQpwN4A457DEAy+WwvwB4Rw5/BMACIcRZkPxQ5QIAEfUAMB7AYLmnUgPgN6EJCSE+gOTBc40s02o57V/EknmGMQIPGTHJQqQho6mq/5M0zq8C8D4RfQ7gczlsCIBfAYAQ4ju5Z9AU0qY6v5TDC4noiBx/BIBzASyR3PEgA3XO5EI5E9LmMwDQSEh+/xnGcVghMEyw22ctXy5jIVX0lwP4KxH1tpAGAXhbCPFgxEjSlqktAaQR0ToAOfIQ0r1CiPkW0mUYw/CQEcNIQznK/x/VJ4goBUAHIcT3AP4MyeVwYwDzIQ/5ENFFAA4KyS//PADXyuFjIG01CUhO5H5NRK3kc1lE1DFUECFEPoBCSLtgPQ3J6VofVgaMG3APgUkWMuSWtsJMIYRietqciFYBqABwTch1qQDeI6JmkFr5LwghjhLRowDekK87hTq3xI8BmEpEawEsArATAIQQ64joIUg74aVA8qB6N4AdGrL2gzSpfBeAZzXOM4wjsLdTJqkhomJI7p0Pei0Lw3gNDxkxDMMwALiHwDAMw8hwD4FhGIYBwAqBYRiGkWGFwDAMwwBghcAwDMPIsEJgGIZhALBCYBiGYWT+H6dommtoOanuAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Plot the learning behavior\n",
    "import pickle\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "with open('./saved_agent/scores.txt', 'rb') as f:\n",
    "    scores = pickle.load(f)\n",
    "\n",
    "plt.plot(scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
